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16.  Abstract 

Interview  data  obtained  from  22 1  general  aviation  (GA)  pilots  are  qualitatively  scored  for  factors  which  influence 
weather-related  decision  making.  Factors  finding  relatively  strong  support  are  (a)  the  specific  type  of  weather  to 
be  faced  (storms,  ice,  visibility,  and  cloud  ceiling  are  of  greatest  concern  to  GA  pilots),  (b)  type  of  flight  (IFR  vs. 
VFR),  (c)  pilot  physiological  state  (primarily  disorientation),  and  (d)  the  inherent  uncertainty  of  weather  and  the 
resultant  cognitive  difficulty  of  understanding  this  uncertainty.  Factors  finding  more  modest  support  are  (a) 
social  and/or  economic  pressures,  and  (b)  impulsive  behavior.  Additionally,  relatively  strong  support  is  found  in 
previously  unpublished  data  for  the  influence  of  mission  goals.  Research  directions,  remediations,  and  the  value 
of  qualitative  analysis  are  discussed. 
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Use  of  Weather  Information  by  General  Aviation  Pilots, 
Part  II,  Qualitative:  Exploring  Factors  Involved  in 
Weather-Related  Decision  Making 


INTRODUCTION 

BACKGROUND  AND  PURPOSE 

While  the  causes  of  aviation  accidents  are  varied  and 
many,  adverse  weather  remains  a  major,  elusive  cause  of 
general  aviation  (GA)  fatalities.  Weather  was  cited  as  causal 
in  only  4%  of  GA  accidents  but  it  accounted  for  12  to 
17%  of  fatalities,  since  about  70%  of  weather-induced 
accidents  prove  fatal  (AO PA,  2005). 

The  U.S.  Federal  Aviation  Administration  has  astated 
current  goal  of  reducing  GA  fatalities  (FAA,  2006).  To 
this  end,  weather  accidents  are  targeted  for  reduction. 

This  calls  for  psychological  research.  Yet,  the  more 
we  learn,  the  more  we  appreciate  how  delicately  the 
complexity  of  weather  interacts  with  the  intricacy  of 
the  human  mind  and  the  pilot’s  unique  task-world.  The 
problem  is  not  simple. 

This  report  is  the  second  in  a  two-part  series,  Part  I 
being  Use  of  weather  information  by  general  aviation  pilots. 
Part  I,  quantitative :  Reported  use  and  value  of  providers 
and  products  (Knecht,  2008).  In  Part  I,  we  examined 
quantifiable  aspects  of  how  an  interviewed  sample  of 
221  GA  pilots  actually  used  the  weather  information 
available  to  them. 

Here  in  Part  II,  the  remaining  interview  data  are  ex¬ 
amined  for  themes — factors  seeming  to  influence  pilot 
weather-related  decision  making.  The  mission  is  to  look 
beyond  quantitative  analysis  to  explore  both  known  and 
novel  weather-related  ideas  worthy  of  further  investigation 
or  actual  intervention. 

METHOD 

Design 

The  original  data  collection  involved  onsite  interviews 
with  GA  pilots.  The  interview  instrument  itself  is  shown 
in  Appendix  A.  In  addition  to  its  quantitative  research 
items,  this  instrument  contained  qualitative  items  de¬ 
signed  to  elicit  free  responses.  Free-responses  are  often 
useful  in  getting  respondents  to  think  beyond  the  rigid 
limits  imposed  by  checkbox  items. 


Participants 

During  July  and  August,  2005,  FAA  staff  conducted 
on-site  interviews  with  230  GA  pilots  at  locations  across  5 
states  (CA,  OK,  ND,  IL,  FL).  Four  venues  were  university- 
based  flight  schools;  the  fifth  was  a  helicopter  training 
course.  Of  these  230  pilots,  221  ultimately  provided  us¬ 
able  data.  Median  pilot  age  was  23  years;  median  flight 
experience  was  245  hours.  Women  comprised  14%  of 
the  sample.  All  were  volunteers  paid  for  their  services  as 
subject  matter  experts. 

Procedure 

Like  the  Part  I  study,  all  responses  were  referenced 
to  a  common  benchmark — a  “standard  flight,”  defined 
as  a  4-hour  flight  through  “weather  serious  enough  to 
challenge  your  skill  level  and  the  aircraft’s  capabilities.” 
This  benchmark  was  crucial  to  the  study.  Without  it, 
pilots  would  have  substituted  their  own  private  defini¬ 
tions  of  “bad  weather,”  making  it  impossible  to  relate 
their  attitudes  and  behavior  to  any  common  real-world 
standard. 

Pilots  were  asked  open-ended  questions  and  Likert- 
scale  items  designed  to  capture  their  thought  processes 
while  dealing  with  adverse  weather.  The  open-ended 
responses  were  then  scored  according  to  a  coding  scheme 
(rubric)  laid  out  in  Appendix  B. 

Rubrics  are  central  to  qualitative  analysis.  A  rubric  is 
a  scoring  methodology,  a  systematic  way  of  defining  a 
set  of  key  factors  well  enough  to  let  a  rater  recognize  and 
tally  specific  instances  within  some  given  text,  speech,  or 
behavior  (Miles  &  Huberman,  1 994).  Multiple  instances 
of  a  single  factor  then  constitute  a  “theme.”  A  theme  can 
be  unique  to  an  individual,  but  we  are  usually  interested 
in  themes  involving  more  than  one  person.  Naturally, 
the  more  respondents  who  mention  a  theme,  the  more 
universal  we  consider  it. 

Rubrics  typically  take  time  to  develop.  They  evolve 
gradually,  within  the  context  of  specific  data,  using  a  process 
of  deduction  and  induction  applied  iteratively.  Deduction 
is  top-down,  logical  reasoning  starting  with  general  prin¬ 
ciples  and  then  looking  for  specific  instances  in  specific 
data.  Induction  is  the  reverse,  the  bottom-up  formation 
of  general  conclusions  after  looking  at  specific  data. 
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Consequently,  three  influences  shaped  the  evolution 
of  our  rubric  (a)  deductive,  (b)  inductive,  and  (c)  sponsor- 
driven.  Cognitive  and  behavioral  psychological  theory 
guided  the  search  for  top-down,  deductive  evidence  of 
certain  themes  in  the  data.  Conversely,  reviewing  specific 
pilot  responses  induced  other  themes  from  the  bottom- 
up.  Finally,  project  sponsors  had  specific  questions  about 
mission-characteristic  effects.  The  rubric  was  amended 
to  assess  those  questions. 

Methodological  issues 

Issues  of  reliability  and  validity  take  on  a  markedly 
different  role  in  qualitative  analysis  than  they  do  in  quan¬ 
titative  analysis.  First,  the  concept  of  “themes”  lends  itself 
less  to  discreteness  and  definability  and  more  to  broadness 
and  generality.  Consequently,  we  typically  recalibrate  our 
priorities  regarding  experimental  error  when  we  go  out 
“fishing  for  factors”  this  way. 

Second,  in  quantitative  analysis  the  focus  is  almost 
always  on  controlling  Type  I  error  (the  misidentification 
of  factors  as  “significant”  which  are  truly  not).  Whereas, 
in  qualitative  analysis  the  focus  is  frequently  on  control¬ 
ling  Type  II  error  (the  failure  to  identify  as  significant 
those  factors  which  truly  are).  This  difference  in  focus 
exists  because  quantitative  analysis  is  about  hypothesis¬ 
testing,  whereas  qualitative  analysis  is  more  often  about 
hypothesis-generation. 

What  we  have  to  understand  is  the  relation  between 
Type  I  and  Type  II  errors.  “Truth”  can  be  thought  of  as  a 
“signal,”  and  signal  detection  theory  says  that  whenever 
we  choose  to  capture  more  signal  (increase  the  hit  rate, 
minimize  Type  II  errors),  this  inevitably  comes  at  the 


expense  of  increased  false  alarms,  namely  increased  Type 
I  error  (Green  &  Swets,  1966).  A  finer-mesh  net  catches 
more  fish — but  also  more  flotsam. 

The  point  is  that  quantitative  and  qualitative  analysis 
are  really  two  methodologies  designed  to  do  different 
things.  Where  quantitative  analysis  tests  hypotheses,  quali¬ 
tative  analysis  can  generate  new  hypotheses.  Researchers 
must  simply  make  clear  when  they  are  engaged  in  each 
type  of  analysis  and  properly  limit  the  interpretation  of 
subsequent  results. 

RESULTS 

Since  the  rubric  was  iteratively  shaped  by  three  influ¬ 
ences  (deductive,  inductive,  sponsor-driven),  the  results 
are  presented  under  similar  headings. 

Deductive  (top-down,  theory-driven)  themes 

Theme  1:  Type  of  weather.  Psychologically,  weather  is 
a  set  of  specific  stimuli.  So,  certainly,  different  types  of 
weather  must  affect  pilot  decisions  differently.  Change 
the  weather,  change  the  decision. 

Following  a  deductive  approach  requires  first  listing 
the  main  weather  factors  pilots  could  discuss.  Next,  using 
their  free- response  answers  (Appendix  A),  the  factors  they 
actually  did  discuss  could  be  scored  and  tabulated. 

An  endnote  [page  9]  clarifies  the  complete  methodol¬ 
ogy  and  exactly  how  to  read  Table  l.1  Briefly,  each  cell 
represents  the  percentage  of  pilots  who  discussed  that 
factor  at  least  once  in  their  interview.  If  a  given  pilot 
mentioned  a  given  factor  more  than  once,  it  still  counted 
only  once.  Note  that  neither  rows  nor  columns  need 


Table  1.  Percentages  of  the  221  pilots  who  referred  at  least  once  to  each  weather  factor  in  their 
interview.  Data  to  the  left  refer  to  preflight.  Data  to  the  right  refer  to  in-flight.  Row  "IR"  data  came 
from  instrument  rated  pilots,  row  "Non-IR"  from  non-IR  pilots.  Row  "Total"  combines  both  IR  and 
non-IR  responses.  "Storms"  mainly  means  thunderstorms,  but  also  tornadoes  +  hurricanes.  “Wind” 
includes  turbulence  +  wind  shear.  Significant  results  are  bold+highlighted  (pz  <  .05  (uncorrected  for 
number  of  comparisons),  1-tailed  for  all  but  "(Non-IR-IR)"  scores,  which  reflect  pz  <  .02,  2-tailed, 
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total  to  100%.  Factors  with  percentages  significantly 
higher  than  average  (p  <  .05)  are  shown  boldfaced  and 
highlighted  in  gray.  Statistical  issues  are  elaborated  in  a 
second  endnote.2 

The  row  labeled  “Total”  represents  instrument-rated 
and  non-instrument-rated  pilots  combined.  Higher- 
than-average  combined-group  preflight  factors  mentioned 
by  pilots  were 


storms 

83% 

ice 

48% 

cloud  ceiling 

46% 

visibility 

57% 

wind 

52% 

In  the  same  row,  farther  to  the  right,  above-average 
combined-group  in-flight  factors  mentioned  were 

•  storms  81% 

•  ice  42% 

•  cloud  ceiling  41% 

•  visibility  49% 

This  gave  us  a  rough  idea  of  what  factors  seemed  to 
mentally  dominate  the  average  pilot  over  the  entire  length 
of  a  flight,  namely  storms,  ice,  ceiling,  and  visibility. 

A  brief  note  about  snow:  Since  the  interviews  were 
conducted  in  the  summer,  it  is  perfectly  natural  to  wonder 
if  the  relatively  low  numbers  for  snow  were  merely  due  to 
the  current  air  temperature.  If  we  had  interviewed  during 
the  winter,  would  snow  have  received  higher  priority?  The 
honest  answer  is  that  we  simply  do  not  know.  There  is 
a  modest  validity  crosscheck  in  the  fact  that  pilots  did 
emphasize  icing.  However,  icing  can  occur  nearly  year- 
round,  especially  carburetor  icing  (and  the  pilots  rarely 
distinguished  between  icing  types  in  their  responses).  So 
the  question  remains  debatable,  and  the  only  valid  (but, 
unfortunately,  cost-prohibitive)  way  to  settle  it  would  be 
to  reconduct  the  study  during  the  winter. 

Theme  2  :  IR  versus  non-IRflying.  Because  instrument¬ 
rated  (IR)  pilots  are  trained  to  fly  in  a  different  physical 
environment  from  non-IR  pilots,  it  made  sense  to  split 
Table  1  in  half  on  this  basis.  The  two  additional  rows 
labeled  “Non-IR”  and  “IR”  also  use  boldface  and  gray  to 
highlight  significantly  above-average  cells  (p  <  .05). 

The  patterns  that  emerged  for  the  two  groups  looked 
fairly  consistent  and  jibed  with  practical  experience. 
Storms,  ice,  ceiling,  and  visibility  are  logically  both 
common  and  potentially  dangerous  to  all  pilots  at  all 
phases  of  flight. 

So  how  did  IR  pilots’  flying  differ  from  non-IR?  The 
“(Non-IR)  -  IR”  row  shows  percentage  dijferencesbetwccn 


the  two  groups.3  Significant  differences  (greater  than  20 
points,  p  <  .02)  emerged  for 

•  Preflight  ice  -32% 

•  In-flight  ice  -28% 

•  In-flight  storms  -21% 

•  In-flight  visibility  +24% 

Negative  numbers  (less  than  zero)  meant  the  factor  was 
bigger  to  IR  pilots.  Numbers  greater  than  zero  meant  the 
factor  was  bigger  to  non-IR  pilots. 

Again,  this  jibed  with  experience.  Non-IR  pilots  must 
avoid  in-flight  restrictions  to  visibility,  while  this  is  much 
less  an  issue  with  IR  pilots. 

Stepping  back  to  get  the  big  picture,  it  was  easy  to  see 
that  instrument  flight  rules  (IFR)  flight  has  a  lot  more 
built-in  ambiguities  than  visual  flight  rules  (VFR)  flight. 
First,  IR  pilots  should  (and  did)  talk  more  about  preflight 
and  in-flight  ice  because  ice  would  not  automatically  pre¬ 
clude  flying,  whereas  it  usually  would  for  non-IR  pilots. 
Second,  in-flight  storms  should  (and  did)  elicit  more 
responses  from  IR  pilots  because  they  would  certainly 
be  more  likely  to  fly  near  (and,  sometimes,  into)  weather 
capable  of  storm  intensity. 

Third,  wind  and  turbulence  loomed  larger  for  everyone 
during  preflight  than  in-flight.  That  might  seem  counter¬ 
intuitive.  But  it  may  simply  have  been  a  technicality.  The 
term  “in-flight”  does  not  technically  include  the  landing 
phase,  whereas  preflight  planning  always  includes  all  phases 
of  flight,  including  landing,  when  wind  and  turbulence 
are  most  likely  to  be  hazardous. 

To  summarize,  in  some  ways  the  environmental  context, 
training,  and  equipment  used  in  instrument  meteorological 
conditions  (IMC)  is  quite  different  from  visual  meteorologi¬ 
cal  conditions  (VMC).  The  basic  hazards  are  the  same  but 
vary  mainly  in  intensity  and  frequency  of  encounter. 

Theme  3:  Social  and  economic  pressure.  Pilots  often  cite 
social  and  economic  pressures  as  reasons  for  taking  chances 
they  would  not  otherwise  take.  For  example,  Rhoda  and 
Pawlak  (1999)  found  that  commercial  airline  pilots  were 
more  likely  to  penetrate  storms  in  terminal  airspace  when 
following  another  aircraft.  In  one  sense,  proximal  pilots 
form  a  “society”  where  one  member  influences  the  risk¬ 
taking  of  others. 

Here,  at  least  48  pilots  reported  external  social  or  busi¬ 
ness-related  pressures  to  fly  in  marginal  weather.  So,  in  an 
informal  sense,  pressure  certainly  was  a  theme. 

Social  psychological  theory  elaborates  this  idea  of  “pres¬ 
sures”  into  better-defined  and  more  measurable  processes 
such  as  social  facilitation,  diffusion  of  responsibility,  social 
assessment  of  risk,  and  obedience  to  authority.  So  did  these 
pilots  report  any  influence  of  these  processes? 
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Task  difficulty 


Figure  1.  The  2x2  matrix  of  possibilities  for  task 
performance  as  defined  by  social  facilitation 
theory.  “+”  means  task  performance  is  enhanced, 
means  it  is  degraded. 

Honestly,  there  was  little  direct  evidence  here  for 
diffusion  of  responsibility  and  social  assessment  of  risk. 
While  we  cannot  rule  them  out,  we  cannot  find  support, 
either.  However,  the  reports  of  “pressure”  were  possibly 
consistent  with  social  facilitation  and/or  obedience  to 
authority. 

Social  facilitation  describes  the  effect  of  an  audience  on 
task  performance  (Zajonc,  1965).  Despite  its  name,  social 
facilitation  is  not  facile.  For  one  thing,  “facilitation”  can 
be  positive  or  negative.  For  another,  outcomes  depend  on 
situational  factors,  for  instance,  task  difficulty  (Figure  1). 
For  example,  experienced  runners  tend  to  run  faster  in  a 
group  than  when  alone  (the  “Simple/ Yes”  cell).  However, 
complex  math  problems  are  easier  to  solve  when  we  are 
by  ourselves  (the  “Hard/No”  cell). 

Does  this  theory  apply  to  weather-related  risk-taking? 
Recall  that  these  pilots  were  predominately  either  students 
or  flight  instructors.  If  students  said  they  felt  pressure 
to  expedite  training,  that  pressure  might  be  caused  by 
social  facilitation. 

However,  one  fact  argues  against  the  social  facilitation 
hypothesis  and  that  is  the  theory  itself.  Weather  flying  is  a 
difficult  task.  Therefore,  as  Figure  1  shows,  the  presence  of 
the  instructor  in  the  cockpit  should  have  inhibited  student 
willingness  to  fly  in  bad  weather,  not  facilitated  it. 

An  alternate,  simpler  economic  explanation  might  be 
more  plausible,  namely  that  flight  training  is  expensive 
and  students  wanted  to  get  through  it  as  fast  as  possible. 


After  all,  many  flight  schools  charge  by  the  hour.  Plus, 
time  spent  training  is  time  away  from  earning  money,  so 
school  carries  a  double  incentive  to  finish  fast. 

Now  what  about  instructors?  Instructors  reported  oc¬ 
casional  pressure  from  both  their  employers  and  students 
to  fly  in  marginal  weather.  While  this  might  be  social 
facilitation,  the  more  straightforward  answer  would  again 
be  economic.  Students  probably  exert  pressure  because 
they  are  trying  to  finish  fast.  Employers  probably  exert 
pressure,  partly  in  response  to  student  pressure  and  partly 
because  the  faster  a  school  moves  students  along,  the  more 
money  it  can  make. 

Finally,  there  was  one  more  alternate  social  hypothesis 
to  consider  here,  namely  obedience  to  authority.  In  his 
classic  experiment,  Milgram  (2004/1974)  demonstrated 
that  otherwise-normal  people  would  administer  suppos¬ 
edly  lethal  electrical  shocks  to  strangers  on  command  from 
an  authority  figure.  If  some  people  would  do  anything 
that  extreme,  might  they  possibly  also  dally  a  bit  with 
bad  weather,  if  pressured  into  it? 

Now,  an  employer  is  definitely  an  authority  figure.  But, 
as  any  teacher  will  confirm,  so  is  the  modern  student. 
This  reflects  the  increasing  tendency  for  schools  to  follow 
a  business  model  where  teachers  have  to  treat  students  as 
customers  (Armstrong,  2003).  And,  as  we  all  know,  the 
customer  is  always  right. 

The  presence  of  passengers  is  another  common  social 
situation  that  could  arguably  influence  risk-taking.  How¬ 
ever,  the  influence  could  conceivably  go  both  ways.  On 
the  one  hand,  passengers  can  be  a  source  of  pressure  to 
start  or  continue  a  flight  into  bad  weather.  On  the  other 
hand,  since  risk  depends  on  what  we  stand  to  lose,  the 
more  lives  that  stand  to  be  lost  (and  the  closer  they  are 
to  us)  the  greater  the  risk  should  be. 

Table  2  was  based  on  two  interview  questions,  “Does 
having  non-family  (Q36,  or)  family  (Q37)  passengers 
affect  your  willingness  to  fly  in  bad  weather?” 

Here,  mostpilots  basicallysaid  “no.”  While  the  reported 
concern  for  family  passengers  was  significantly  greater 
(X-  =  32.4  (3 ),p<  .0000 1 ),  only  33%  ((27+29+ 1 8)/22 1 ) 
reported  more  than  “a  little  bit”  of  concern  over  having 
non-family  passengers,  as  compared  to  47%  for  family 
passengers. 

Beyond  the  palpable  conclusion  that  family  trumps 
non-family,  these  results  were  hard  to  interpret.  Could 


Table  2.  Influence  of  passengers  (response  frequencies) 
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the  majority  of  passengers  be  quite  so  persuasive?  Or 
could  the  majority  of  pilots  believe  in  their  own  skill  and 
luck  so  profoundly  that  having  others  on  board  added 
absolutely  nothing  to  their  perception  of  risk? 

Both  those  conclusions  are  absurd.  What  is  more 
believable  is  that  multiple  motivations  are  confounded 
here,  some  facilitative,  some  inhibitory  And  the  net 
effect  that  any  given  passenger  must  have  on  a  pilot’s 
willingness  to  take  weather- related  risks  probably  depends 
on  the  specific  pilot  plus  the  specific  passenger  plus  the 
specific  circumstances.  A  boss  pressuring  you  to  get  to 
a  meeting  on  time  is  different  from  taking  your  child¬ 
hood  friend  out  to  practice  touch-and-goes.  Individuals 
matter,  relationships  matter,  the  flight  mission  matters, 
and  so  forth. 

To  wrap  this  up,  both  social  and  economic  circum¬ 
stances  can  obviously  influence  weather-related  behavior. 
But  there  is  a  lot  more  to  it  than  meets  the  eye.  First,  some 
factors  may  increase  risk-taking,  others  decrease  it.  Second, 
situational  details  matter,  and  broad  questions  such  as  the 
ones  asked  here  will  not  tease  apart  all  the  causal  intrica¬ 
cies.  Example:  Question  17h  was  worded  “Have  social 
or  business  pressures  ever  influenced  your  GA  go/no-go 
weather  decision?”  While  not  a  terrible  question,  future 
interviews  should  try  to  devise  more  specific,  theory-based 
questions  better  able  to  differentiate  facilitation  from 
inhibition  and  able  to  distinguish  social  inhibition  from 
obedience,  economic,  and  other  motivations. 

Theme  4:  Pilot  physiological  state.  Factors  such  as 
fatigue,  hypoxia,  and  spatial  disorientation  affect  weather- 
related  decision  making  (Taneja,  2002).  Interview  time 
constraints  prevented  full  exploration  of  all  these  factors, 
but  we  did  sample  disorientation.  When  asked,  “How 
many  times  have  you  become  so  disoriented  that  you  had 
to  land  or  call  ATC  for  assistance  in  determining  your 
location?”,  22  pilots  responded  (Q3 1 ,  average  1 . 1  episodes 
per  pilot).  Additionally,  when  asked,  “How  many  times 
have  you  become  so  disoriented  after  entering  IMC  that 
you  had  difficulty  in  maintaining  aircraft  control?”,  35 
pilots  responded  (Q33,  average  1.5  episodes  per  pilot). 

Given  how  relatively  low-hour  most  of  these  pilots  were, 
these  numbers  were  surprisingly  high.  Fortunately,  of  pi¬ 
lots  who  admitted  to  disorientation,  only  32%  responded 
to  both  questions.  Put  another  way,  up  to  two-thirds  of 
those  who  did  experience  serious  disorientation  may  have 
undergone  one-trial  learning.  Powerful  experiences  tend 
to  have  that  effect. 

Of  course,  the  nagging  facts  remained  that  (a)  about 
20%  of  mostly  low-hour  pilots  reported  getting  seriously 
disoriented  at  least  once  and  (b)  there  are  more  physiologi¬ 
cal  effects  than  just  disorientation.  It  seems  that  serious 
physiological  disruptions  may  be  more  common  than  we 


thought.  Moreover,  these  people,  even  if  they  did  learn 
effectively,  learned  the  hard  and  dangerous  way.  And  the 
handful  who  went  through  the  experience  more  than 
once  may  constitute  a  target  group  in  need  of  training 
or  intervention. 

Theme  5:  Impulsivity.  Nearly  every  adult  acts  impul¬ 
sively  at  least  once  in  their  adult  life.  Pilots  are  specifically 
trained  to  minimize  impulsive  behavior.  Do  they? 

Fortunately  for  this  analysis,  more  than  a  few  pilots 
felt  comfortable  enough  to  admit  to  various  kinds  of 
impulsive  or  accidental  rule-breaking  behaviors  such  as 
taking  off  with  no  weather  planning  at  all  (Q17g,  40  af¬ 
firmative  responses)  or  flying  into  IMC  without  proper 
rating  or  aircraft  (Q32,  35  affirmatives). 

This  is  evidence  we  should  conservatively  call  “un¬ 
intentional  impulsivity”  because  none  of  the  questions 
overtly  asked  about  intentions  (realistically,  how  many 
pilots  would  have  admitted,  in  writing,  to  an  FAA  official, 
that  they  had  intentionally  broken  any  rule?) 

So,  these  answers  did  show  that  pilots  do  act  on  im¬ 
pulse  once  in  a  blue  moon.  While  the  vast  majority  of  this 
behavior  obviously  did  not  result  in  serious  consequences 
in  the  sense  of  having  hurt  someone,  the  potential  seri¬ 
ousness  was  impossible  to  gauge  because  the  sample  was 
biased.  It  contained  no  one  who  had  ever  been  killed  as 
a  result  of  impulsive  behavior.  If  such  pilots  could  talk, 
they  might  judge  impulsivity  far  more  harshly  than  either 
we  or  these  pilots  did. 

Inductive  (bottom-up,  emergent)  themes 

Induction  is  the  opposite  of  deduction.  In  examining 
the  data,  patterns  and  themes  emerge,  induced  by  the 
evidence. 

Theme  6:  The  uncertainty  of  weather.  A  strong  theme 
gradually  materialized  after  multiple  reviews  of  the  data. 
It  started  with  the  broad  idea  of  uncertainty.  Pilots  made 
at  least  30  general  and  21  specific  remarks  concerning 
the  uncertainty  of  weather.  So  how  could  this  effect  of 
uncertainty  be  best  understood? 

There  are  at  least  four  main  influences  that  drive  the 
uncertainty  of  weather  prediction 

•  Weather  is  chaotic 

•  Most  of  us  have  difficulty  understanding  prob¬ 
abilities 

•  Weather  forecasts  are  biased  towards  false  alarms 

•  Weather  risk  increases  with  flight  length 

First,  weather  truly  is  hard  to  predict.  Compared 
to  simple,  orderly  probabilistic  systems  like  cards  or 
dice,  the  earth’s  surface,  oceans,  and  atmosphere  form 
a  complex,  chaotic  system  in  the  strictest  mathematical 
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sense  (Gleick,  1987).  The  mathematical  models  that 
forecasters  use  to  make  predictions  are,  by  definition, 
simplifications  which  contain  error.  Over  time,  this  error 
gets  amplified  and,  inevitably,  the  forecast  becomes  less 
and  less  accurate.  Anything  over  24  hours  is  literally  a 
“long  time”  in  forecasting. 

Second,  estimating  and  understanding  probabilities  is 
something  most  of  us  are  not  terribly  good  at,  particu¬ 
larly  with  rare  events.  Kahneman,  Slovic,  and  Tversky 
(1982)  and  many  others  have  shown  that  people  often 
undere  stimate  the  occurrence  of  likely  events  that  happen 
to  be  commonplace  while  overestimating  the  occurrence 
of  much  less  likely  events  that  merely  happen  to  be  more 
dramatic. The  chance  ofbeingkilledbylightning is  agood 
example.  Many  of  us  would  think  this  is  fairly  common, 
yet  lightning  claims  just  about  200  Americans  each  year 
(National  Weather  Service,  2006a).  This  is  a  small  figure 
compared  to  the  44,000  typically  taken  by  car  accidents 
(Bureau  of  Transportation  Statistics,  2006).  It  is  as  if  we 
have  a  built-in  tendency  to  confuse  high  drama  with 
high  likelihood. 

A  third  factor  driving  the  uncertainty  ofweather  predic¬ 
tion  is  the  fact  that  weather  forecasts  are  biased  towards 
false  alarms.  From  a  static  observer-centered  point  of 
view — the  point  of  view  of  any  given  observer  standing 
still  on  the  ground — weather  forecasts  tend  to  overpre¬ 
dict  the  chance  of  bad  weather.  This  is  a  rather  obscure 
mathematical  artifact  which  requires  some  explanation. 

Bad  weather  is  something  we  want  to  detect.  It  is  a 
“signal”  buried  in  “probability  noise.”  This  lets  us  draw 
the  four  outcomes  defined  by  signal  detection  theory 
(Figure  2). 

A  “false  alarm”  happens  when  bad  weather  is  predicted, 
but  the  actual  weather  turns  out  good  (or  at  least  better 
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Figure  2.  The  2x2  matrix  of  possibilities  for 
weather  prediction,  as  defined  by  signal  detection 
theory. 


than  predicted).  A  “miss”  is  when  good  weather  is  pre¬ 
dicted,  but  the  actual  weather  turns  out  bad  (or  at  least 
worse  than  predicted). 

Misses  and  false  alarms  are  our  two  types  of  mistakes. 
Too  many  of  either  and  we  start  mistrusting  the  weather 
forecast. 

Weather  forecasters  do  not  want  us  to  mistrust  the 
forecast.  But  they  also  know  that  people  dislike  being 
surprised  by  unexpected  bad  weather  far  more  than 
they  do  being  surprised  by  unexpected  good  weather. 
So,  forecasts  are  typically  conservative  and  effectively 
overpredict  bad  weather.  In  fact,  when  the  forecast  says 
“20%  chance  of  rain,”  all  it  really  means  is  that,  within 
a  rather  large,  pre-specified  geographical  area  during  a 
pre-specified  time  period,  there  is  a  20%  chance  of  at  least 
.01  inches  of  rain  falling  somewhere  (National  Weather 
Service,  2006b). 

This  way  of  defining  bad  weather  automatically  leads 
to  likelihood  overestimation.  The  easiest  way  to  under¬ 
stand  is  to  think  of  weather  as  a  shotgun.  If  you  shoot  at 
a  cereal  box  1 00  yards  away,  you  may  have  a  20%  chance 
of  hitting  it.  But  the  chance  of  hitting  a  flea  on  top  of 
that  box  is  much  smaller.  Smaller  size — less  chance  of 
getting  hit.  Now,  just  imagine  you  are  the  “flea”  and  the 
“cereal  box”  is  the  size  of  Connecticut.  The  idea  is  that 
weather  probabilities  apply  to  large  geographic  regions, 
not  to  one,  tiny  person.  Your  chance  of  personally  being 
hit  by  something  is  typically  much  less  than  the  forecast 
suggests. 

Unfortunately,  this  bias  has  a  great  effect  on  aviation. 
Given  this  built-in  tendency  for  false  alarms,  pilots  start 
expectingxhe  actual  weather  to  be  less  severe  than  predicted. 
And  then,  like  the  boy  who  cried  “Wolf,”  along  comes 
a  time  when  the  actual  weather  turns  out  far  worse  than 
predicted.  Given  enough  time,  this  is  statistically  almost 
certain  to  happen. 

The  fourth  influence  driving  the  uncertainty  ofweather 
prediction  is  yet  another  statistical  quirk,  one  biased  in 
the  opposite  direction  to  the  one  just  discussed.  Longer 
flights  usually  involve  more  bad  weather  than  short  flights. 
The  reason  is  based  in  statistical  mechanics  (Knecht, 
2000)  but  is  easy  to  understand:  The  farther  you  fly, 
the  more  airspace  you  plow  through.  The  more  airspace 
you  plow  through,  the  more  chance  you  have  of  running 
into  bad  weather  somewhere.  It  is  like  walking  through 
a  minefield.  The  farther  you  travel,  the  more  likely  you 
are  to  step  on  something  bad. 

So,  to  summarize,  uncertainty  is  rampant  in  weather 
forecasting.  First,  weather  is  chaotic,  hence,  truly  hard 
to  predict.  Second,  we  innately  tend  to  overestimate  the 
likelihoods  of  some  events  while  underestimating  others. 
Third,  weather  forecasts  are  biased  to  overpredict  bad 
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weather  for  stationary  observers.  Finally,  an  opposite  bias 
also  occurs:  that  longer  flights  run  greater  weather  risk. 

With  so  many  deep  and  contradictory  forces  at  work, 
is  it  any  wonder  why  weather  confuses  us? 

Fortunately,  many  of  the  pilots  we  interviewed  openly 
acknowledged  this  issue  of  uncertainty.  At  least  86  explicit 
references  were  made  to  specifics  such  as  “expecting  the 
unexpected”  and  of  having  preconsidered  options  such  as 
diverting  to  alternate  airports  or  driving  instead  of  flying. 
This  showed  an  appreciation  of  the  underlying  problem 
and  a  healthy  evolution  of  strategies  to  combat  it. 

Sponsor-driven  questions 

This  work  was  partially  tasked  to  address  an  additional, 
sponsor-driven  question,  namely: 

Theme  7:  To  what  extent  do  mission  goals  influence 
weather  analysis  and  decision  making?  Unfortunately, 
there  was  little  evidence  in  this  study  to  illuminate  that 
theme  beyond  discussion  ofTheme  3.  Since  the  current 
interview  questions  specified  neither  a  range  of  weather 
nor  of  missions,  it  was  no  surprise  that  mission  goals  did 
not  come  up  spontaneously  as  a  theme. 

Fortunately,  the  basic  question  was  addressed  by  prior 
work.  In  unpublished  data  gathered  by  Knecht,  Flarris, 
and  Shappell  (2005),  105  GA  pilots  were  asked  about 
their  willingness  to  fly  in  assigned  visibilities  ranging  from 
1-5  smand  ceilings  of  1000-2000’.  One  debrief  question 
was,  “Ifyour  flight  mission  had  been  critical  (for  example, 
delivering  a  human  heart  for  surgery),  how  much  would 
that  change  your  willingness  to  take  off/continue?” 

Given  that  hypothetical  situation,  virtually  everyone 
indicated  a  strong  willingness  to  fly  in  the  visibility/ ceil¬ 
ing  combination  to  which  they  had  been  assigned.  This 
was  particularly  striking,  given  that  many  had  actually 
chosen  not  to  fly,  even  when  offered  $200  to  do  exactly 
that.  Now,  while  respondents  admittedly  may  have  been 
merely  trying  to  cast  themselves  in  a  good  light,  the  chances 
are  good  that  these  answers  really  did  reflect  a  genuine, 
altruistic  human  concern  for  fellow  citizens. 

If  so,  then  this  addresses  the  influence  of  flight  mis¬ 
sion.  Given  an  extremely  critical  mission,  nearly  all  pilots 
will  fly  into  some  fairly  bad  weather.  Conversely,  given 
severe  weather  coupled  with  an  unimportant  mission, 
nearly  no  one  will. 

That  defines  two  ends  of  a  continuum.  But  remem¬ 
ber  that  this  is  a  multidimensional  decision  landscape 
with  many  possible  weather  characteristics  and  mis¬ 
sion  characteristics.  As  is  usual  in  such  cases,  details 
are  critical,  and  every  scenario  ends  up  having  to  be 
examined  separately. 


DISCUSSION  AND  CONCLUSIONS 

This  research  was  the  second  in  a  two-part  series. 
Part  I  looked  at  quantitative  data  from  221  GA  pilot 
interviews,  categorizing  the  types  of  weather  informa¬ 
tion  pilots  could  use  versus  the  ones  they  say  they  do 
use  (Knecht,  2008). 

Flere  in  Part  II,  the  emphasis  was  qualitative.  Using  the 
same  data  set,  we  looked  for  further  evidence  of  weather- 
related  factors  that  influence  pilot  decision  making  and 
which  were  either  logically  deducible  facts  or  which  could 
be  arrived  at  inductively  by  scoring  and  tallying  pilots’ 
free-response  items.  The  underlying  goal  was  hypothesis- 
generation  for  possible  future  exploration. 

The  following  factors  were  fairly  strongly  sup¬ 
ported: 

1.  “Short  list”  of  weather  types  uppermost  in 

pilots’  minds 

a.  storms 

b.  ice 

c.  deteriorating  visibility 

d.  lowering  cloud  ceilings 

2.  Specific  pilot  factors  influencing  the  “short  list” 

a.  instrument  rating 

b.  experience  with  weather 

3.  Spatial  disorientation 

4.  Uncertainty 

a.  forecast  accuracy  decreases  as  lookahead  time 
increases 

b.  forecast  probabilities  are  greatly  misunderstood 
by  most  people 

c.  forecasts  routinely  overpredict  bad  weather  for 
stationary  observers 

d.  weather  risk  increases  as  flight  length  in¬ 
creases 

The  following  factors  found  modest  support: 

5.  Social  and  economic  pressures  can  increase  risk¬ 
taking 

6.  Strictly  impulsive  behavior  is  rare  but  does 
exist 

Finally,  there  was  narrowly  focused  (but  fairly  strong) 
support  found  in  previously  unpublished  data  for  the 
influence  of: 

7.  mission  goals 

Factors  1  and  2  receive  extensive  intellectual  training 
in  GA  flight  training.  But  one  important  thing  new 
pilots  typically  lack  is  the  “fear  factor”  which  can  only 
be  learned  from  the  physical  and  emotional  experience  of 
storms,  ice,  and  inadequate  visibility. 
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Factor  3  concepts  are  also  covered  in  flight  train¬ 
ing.  But,  again,  unless  pilots  have  spent  time  in  special 
full-motion  disorientation  training  simulators,  physi¬ 
ological  effects  will  probably  end  up  being  learned  the 
“old-fashioned”  way. 

Most  of  Factor  4  receives  little  emphasis.  We  will  return 
to  it  in  a  moment. 

Factor  5  is  one  everybody  acknowledges  but  no¬ 
body  does  much  about  because  it  is  about  psychol¬ 
ogy,  not  strictly  about  pilot  training.  To  resist  social 
and  economic  pressures  takes  knowledge,  wisdom, 
and  character  as  pilot-in-command.  Flight  schools 
cannot  teach  wisdom.  That  is  a  lifelong  process.  But 
they  can  offer  guidance,  encourage  pilots  to  develop 
personal  minima,  and  share  personal  accounts  of 
weather  encounters,  both  their  own  and  those  of  pilots 
they  know.  Personalization  can  help  make  the  abstract 
more  concrete. 

Factor  6  is  the  reason  there  are  few  “old,  bold  pilots.” 
Impulsivity  catches  up  with  them.  Impulsivity  is  aviations 
crabgrass — we  can  control  it  but  can  never  completely 
get  rid  of  it.  Our  culture  values  daring  behavior.  Flying  is 
synonymous  with  daring.  So  how  do  we  honor  our  culture 
and  still  teach  the  disciple  that  discretion  is  usually  the 
better  part  of  valor?  Maybe  the  answer  is  to  teach  that 
there  is  no  honor  in  unnecessary  risk. 

Little  direct  support  was  found  for  Factor  7  in  these 
data  (although  it  was  in  other,  unpublished  data).  Given 
the  right  mission,  pilots  will  take  on  considerable  risk. 
But  the  relation  is  complex  and  the  details  of  the  mission 
and  the  weather  both  matter. 

Most  of  these  factors  are  well-known  and  our  rediscov¬ 
ery  of  them  far  from  ground-breaking.  But  they  are  real, 
and  our  focus  on  them  has  to  remain  relentless. 

Factor  4  is  perhaps  the  most  intriguing  and  novel  part 
of  this  report.  We  all  know  weather  is  uncertain,  but  we 
rarely  sit  down  and  think  exactly  why  it  is  uncertain  and 
why  it  is  that  the  human  mind  has  such  difficulty  with 
this  particular  style  of  uncertainty. 

The  uncertainty  of  weather  starts  with  the  physical 
world.  Weather  is  complex  and  chaotic  in  the  true  math¬ 
ematical  sense — impossible  to  fully  model,  impossible  to 
fully  predict.  That  leaves  probability  as  the  best  we  can 
do  to  express  how  weather  behaves. 

Probabilities  segue  into  psychology.  By  nature,  the  hu¬ 
man  mind  hates  uncertainty.  The  greater  the  uncertainty, 
the  more  uncomfortable  we  are  with  it.  This  discomfort 
is  deep-rooted,  genetic,  and  probably  related  to  survival 
of  the  species.  We  seem  to  have  biases.  We  overestimate 
some  odds  and  underestimate  others.  Why?  Because,  in 


the  still  of  the  night,  there  may  be  tigers.  And,  those  of 
our  ancestors  biased  to  worry  a  little  too  much  about  tigers 
may  have  ended  up  living  longer  than  those  who  worried 
too  little.  So,  these  mental  biases  may  be  the  remnants 
of  tendencies  that,  under  more  primitive  circumstances, 
conveyed  survival  advantage.  But,  here  in  the  modern 
world,  a  side  effect  is  that  we  overestimate  the  average 
severity  of  weather  forecasts. 

Intuition  tells  us  that  bad  weather  is  less  likely  to  hap¬ 
pen  than  the  forecast  says — tfyou  stand  in  one  spot.  Now, 
statistical  mechanics  shows  that  intuition  was  right. 

Finally,  as  if  all  this  were  not  enough,  we  now  know 
that  the  opposite  is  true,  too,  provided  we  are  wotstanding 
in  one  spot.  The  farther  we  fly,  the  more  adverse  weather 
we’re  likely  to  encounter. 

We  now  have  a  clearer  understanding  of  why  weather 
forecasting  is  inherently  problematic  and  will  never  be 
trivial.  This  is  no  cause  for  alarm  or  discouragement, 
but  pilots  do  need  to  come  to  the  same  kind  of  clear 
understanding.  Exactly  how  to  do  that  will  occupy  us 
for  some  time  to  come. 

Suggestions  for  further  study  and/or  intervention 

Uncertainty  is  the  pilot’s  greatest  enemy.  What  we 
know  and  know  of,  we  can  either  cope  with  or  avoid.  Five 
healthy,  positive  weather-related  responses  are: 

1.  Use  and  understand  the  modern  weather  products. 

2.  Expect  the  unexpected. 

3.  Always  have  multiple,  workable  options  thought  out 
ahead  of  time. 

4.  Do  not  wait  to  learn  about  weather  the  hard  way. 

3.  There  is  no  honor  in  unnecessary  risk. 

We  can  teach  weather  skills.  The  utility  of  low-cost, 
PC-based  weather  training  comes  to  mind,  as  well  as  low- 
cost  cockpit  weather  information  devices  (and  we  need 
to  stress  that  the  Flight  Service  Station  is  the  lowest-cost 
“device”  of  all). 

Ultimately,  we  need  even  more  reliable,  more  effec¬ 
tive  weather  forecasts.  In  the  meantime,  pilots  need  to 
seek  out  the  excellent  preflight  preparation  products 
already  in  place  (e.g.,  www.aviationweather.gov).  Pilots 
need  to  understand  clearly  what  those  mean  and  how  to 
use  them.  Right  now,  these  weather  providers  are  new, 
and  we  are  all  still  learning  how  to  make  the  most  of 
them.  Familiarity  and  understanding  will  spontaneously 
increase  with  use,  so  that  aspect  is  self-correcting.  In 
the  meantime,  human  factors  study  of  the  graphic  user 
interfaces  might  be  of  service  to  the  National  Weather 
Service  and  should  be  considered. 


ENDNOTES 

'To  understand  Table  1,  first  notice  the  highlighted  “Total” 
on  the  left-hand  side.  Now  follow  the  percentage  of  pilots  who 
mentioned  weather  factors  in  that  row. 


Total 

83 

5 

7 

48 

5 

46 

57 

52 

Each  number  represents  the  percentage  of  pilots  who 
“voted”  for  that  weather  factor  by  mentioning  it  in  their 
interview.  Each  cell  percentage  was  calculated  as 


#  of  pilots  who  mentioned  this  factor  at  least  once 
total  #  pilots  (N=221 ) 

If  a  pilot  mentioned  a  given  factor  more  than  once,  it 
still  only  counted  as  one  “vote.”  Therefore,  the  smallest 
percentage  a  given  factor  could  get  was  0%  and  the  larg¬ 
est,  100%.  Notice  that  rows  do  not  add  up  to  100%.  We 
do  not  expect  them  to,  because  each  factor  was  separate 
and  based  on  its  own  separate  tally  of  votes. 

The  row  at  left  labeled  “Total”  considered  instrument¬ 
rated  and  non-instrument-rated  pilots  together  as  a  total 
group.  Eligher-than-average  percentages  (highlighted, 
bold)  that  emerged  as  preflight  factors  were,  storms  (83%), 
ice  (48%),  cloud  ceiling  (46%),  visibility  (57%),  and 
wind  (52%). 

In  the  same  row,  farther  to  the  right,  similar  numbers 
emerged  for  in-flight  factors — ones  supposedly  influencing 
pilots’  go/no-go  weather  decisions  while  in-flight. 


11 

81 

5 

4 

42 

5 

41 

49 

38 

Elere,  higher-than-average  percentages  emerging  as 
in-flight  factors  were  storms  (81%),  ice  (42%),  cloud 
ceiling  (41%),  and  visibility  (49%). 

Instrument  flying  is  different  from  VFR  flight,  so  we 
sorted  pilots  by  instrument  rating.  These  percentages 
form  the  two  rows  labeled  (at  left)  “Non-IR”  and  “IR.” 
In  these  two  rows,  highlighting  and  boldface  again  mark 
the  most  frequently  mentioned  factors. 

The  limitations  of  Table  1  must  be  explained.  Over¬ 
all,  what  we  were  trying  to  do  was  get  a  sense  of  which 
weather  factors  were  most  important  to  pilots.  Since  each 
cell  represented  the  percentage  of  pilots  who  mentioned 
a  particular  factor  at  least  once  during  their  interview, 
we  assumed  that  the  more  pilots  who  mentioned  that 
factor,  the  more  important  the  factor  was  likely  to  be  in 
their  decision  making.  Thus,  the  higher  numbers  prob¬ 
ably  reflected  the  more  important  weather  factors.  The 
question  then  became  how  big  did  the  numbers  have  to 
be  to  be  considered  reliable? 


The  first  step  was  to  compare  each  cell  in  a  row  with 
its  corresponding  row  mean.  This  would  determine  which 
weather  factors  were  above  average.  To  establish  reliabil¬ 
ity,  in  each  of  the  “Total,”  “non-IR,”  and  “IR”  rows,  a 
standard  error  of  proportion  (SEMrow)  was  calculated  to 
estimate  the  stability  of  that  row  mean  (Ferguson,  1971, 
ch.  12).  Then,  each  individual  cell  score  was  evaluated 
against  (row  mean  +  1.65  SEM  ,  ~  “p  <  .05,”  1-tailed). 
Note  that  this  is  not  the  same  as  finding  “the  upper  5% 
of  cell  values.”  It  represents  finding  which  weather  fac¬ 
tors  seem  reliably  more  important  than  average.  Thus  we 
expect  to  see  about  half  the  factors  identified,  which  is 
the  case  in  Table  1 .  Statistical  purists  may  criticize  this 
method,  but  we  need  to  keep  in  mind  the  largely  qualita¬ 
tive,  exploratory  nature  of  this  study. 

The  second  thing  we  wanted  to  do  was  compare  non¬ 
instrument-rated  pilots  with  instrument-rated  pilots 
to  see  which  weather  factors  seemed  more  important 
to  which  group.  This  method  was  considerably  more 
precise.  It  involved  comparing  cells  by  column,  between 
“Non-IR”  and  “IR”  for  each  weather  factor.  To  do  that, 
a  separate  SEMcolumn  was  calculated  for  each  “(Non-IR) 
-  IR”  difference  score  (because  SEM  is  a  function  of  both 
proportions),  and  that  was  used  to  do  z- tests  (Ferguson, 
1971,  ch.  12).  In  this  case,  we  elected  to  use  a  much 
more  stringent  criterion  for  reliability,  p  <  .  02,  2-tailed, 
to  correct  for  multiple  comparisons. 

To  summarize,  the  primary  limitation  here  was  that  the 
method  was  extremely  lax  about  the  “row  conclusions,” 
while  being  moderately  strict  about  “column  conclusions.  ” 
Therefore,  the  reader  is  strongly  cautioned  to  keep  this 
in  mind,  as  appropriate  to  the  circumstance. 

2In  Table  1  (and  throughout  this  report),  most  numbers 
presented  are  not  necessarily  statistically  significant.  This 
involves  a  technical  issue  having  to  do  with  experiment- 
wise  error.  When  many  statistical  results  are  reported  in 
one  study,  there  is  a  good  chance  that  at  least  some  of 
those  “significant”  results  will  be  false.  Elowever,  if  one 
uses  standard  procedure  to  statistically  correct  for  this 
(e.g.,  the  Bonferroni  correction),  then  one  lands  squarely 
in  territory  where  nothing  is  “significant.”  This  is  one 
reason  why,  in  exploratory  research  such  as  this — and, 
particularly  in  qualitative  research — results  are  typically 
presented  “as  is,”  with  no  deep  statistical  analysis,  and 
no  correction  for  experimentwise  error. 

3 At  first  glance,  it  might  seem  appropriate  to  do  analy¬ 
sis  of  variance  (ANOVA)  on  these  numbers.  But  many 
variables  (e.g.  .storms,  rain,  hail)  are  correlated  and,  to  an 
unknown  degree,  violating  the  assumptions  of  ANOVA 
and  rendering  it  inappropriate. 
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APPENDIX  A 
Pilot  Interview 


SME# 


1-  Age _ 

2.  Gender  (male _ ,  female _ ) 

3.  Primary  occupation _ 

4.  Other  current  occupation(s) _ 

5.  Past  occupations(s)  related  to  aviation _ 

6.  Certificates  and  ratings  (check  each  that  applies) 


Sport 

Airplane  Single-Engine 

Recreational 

Airplane  Multiengine 

Private 

Rotorcraft 

Commercial 

Balloon 

ATP 

Airship 

Instrument 

Glider 

Flight  Instructor 

Powered-Lift 

7.  Type  of  flying  you  do  (to  the  nearest  10  percent,  for  example,  recreational  20%) 

recreational _  business _  corporate _  commercial _ (these  should  add  to  100%) 

For  questions  below,  “general  aviation”  (GA)  means  “any  small  aircraft  not  flying  for  hire.” 

8.  Your  total  GA  flight  hours  (best  guess) _ Total  hours  in  last  90  days _ 

9.  Do  you  own  your  own  GA  aircraft,  either  by  yourself  or  as  a  member  of  a  partnership?  (Y  /  N) 

10.  Type(s)  of  GA  aircraft  usually  flown: _ 

11.  Your  normal  personal  minimum  for  GA  VFR  visibility  _ statute  miles 

12.  Your  normal  personal  minimum  for  GA  VFR  cloud  ceiling  _ feet  AGL 

For  questions  below,  if  you’re  not  a  U.S.  citizen,  use  “country”  instead  of  “state” 

13.  Current  home  state  (legal  residence) _ 

14.  Approximate  percentage  of  time  you’ve  flown  GA  in  your  home  state _ %  versus 

outside  your  home  state _ %  (est.:  add  up  to  100%) 

15.  State(s)  where  you  received  GA  pilot  training _ 

16.  States  where  you’ve  flown  GA  (put  a  check  mark  s/  in  each  state  name  below) 


If  your  flying  has  been  largely 
outside  of  the  USA,  please  list 
below  the  countries  in  which  you 
regularly  fly  and  the  percentages 
of  time  spent  in  each  (estimates): 


Country 

%  time 

A-l 


This  is  a  study  about  how  GA  pilots  use  weather  information.  Please  bear  in  mind  these  things: 

A.  We  already  know  the  “textbook  answers”  for  how  pilots  are  supposed  to  use  weather  information.  What  we 
need  to  know  is  how  real  pilots  are  using  real  weather  information  in  the  real  world. 

B.  Your  responses  are  strictly  anonymous  and  confidential. 

C.  In  the  next  section  we’ll  refer  to  “cross-country  flights.”  That  may  mean  different  things  to  different 
people.  So  define  “cross-country”  as:  1)  Non-local  airport,  far  enough  away  that  the  weather  could  surprise 
you. 

D.  “Bad”  weather  can  also  mean  different  things.  So  define  it  as:  Weather  serious  enough  to  challenge  vour 
skill  level  and  the  aircraft’s  capabilities. 


SECTION  TWO:  CROSS-COUNTRY,  BAD  WEATHER  GA  FLIGHT 

17.  This  question  will  ask  details  about  how  you  get  a  PREFLIGHT  weather  briefing  for  CROSS-COUNTRY, 
GA  FLIGHT  when  you  ANTICIPATE  BAD  WEATHER.  Use  the  definitions  of  “cross  country”  and  “bad 
weather”  from  above  in  forming  your  responses. 

a.  When  do  you  start  planning  such  a  flight?  (for  example,  the  day  before,  the  morning  of,  etc.) 

b.  Where  do  you  start  researching  the  weather?  (e.g.,  at  home?  At  the  airfield?) 

(Below,  a  weather  “product”  is  a  single  report  like  a  METAR,  TAF,  ASOS,  or  AWOS.  A 
“provider”  is  an  organization  like  the  FSS  that  bundles  individual  products  together  to 
give  a  comprehensive  wx  outlook) 

c.  List  the  main  weather  information  provider(s)  you  consult.  List  the  main  products  you  use  from 
each  provider.  What  relative  importance  do  you  give  to  these  products?  (write  “1”  by  the  most 
important  product,  “2”  by  the  second-most  important  product,  etc. 


d.  About  how  many  minutes  does  usually  it  take  to  finalize  your  bad-weather  GA  plan? _ 

e.  List  the  major  weather  factors  that  would  immediately  trigger  a  no-go  decision  before  takeoff. 


f.  What  weather  factors  would  lead  you  to  divert  a  flight  in  progress? 


g.  Is  there  any  time  you  anticipated  bad  weather  but  took  off  without  planning  for  it?  If  so,  describe  it 
briefly.  Remember — this  is  100%  anonymous,  so  do  NOT  name  names  of  individuals  involved. 


A-2 


h.  Have  social  or  business  pressures  ever  influenced  your  GA  go/no-go  weather  decision?  (For 

example,  have  you  ever  made  a  risky  flight  on  a  dare,  or  has  a  boss  ever  pressured  you  into  flying 
against  your  better  judgment?).  If  so,  describe  it,  taking  care  not  to  name  names. 


i.  In  plain  words,  describe  what  goes  through  your  mind  in  planning  for  bad-weather,  cross-country 
GA  flight. 


j.  Briefly,  how  does  your  good-weather  planning  differ  from  your  bad-weather  planning? 


k.  If  there  were  one  thing  you’d  like  to  see  improved  about  weather  information,  what  would  it  be? 


A-3 


SECTION  3:  CROSS-COUNTRY,  BAD-WEATHER  INFO.  SOURCES  (IN-DEPTH  REPORT) 
(As  before,  a  “product'’  is  a  single  report.  A  “provider”  combines  products  to  give  a  big  picture) 


18.  Evaluate  the  top  5  pre  flight  weather  providers  you  use  most  to  plan  a  cross-country,  bad-weather  flight. 

Rank:  Using  the  l-to-5  scale  below,  rank  ONLY  your  5  most-used  providers  (leave  others 
blank). 


1 

2 

3 

4 

5 

most-used 

above  average 

average 

below  average 

least-used 

b.  Value:  Using  the  l-to-5  scale  below,  rate  the  information  value  of  each  of  those  top  5  choices. 


1 

2 

3 

4 

5 

excellent 

above  average 

average 

below  average 

poor 

c.  %:  Estimate  the  percentage  of  cross-country,  bad-wx  flights  you  use  each  of  these  top  5  providers 
on.  (NOTE:  In  18c,  19c,  and  20c,  the  percentages  do  NOT  have  to  add  up  to  100%) 


d.  Minutes:  Estimate  the  average  number  of  minutes  spent  on  each  of  the  5  during  bad-wx  prefli 


Rank 

Value 

Provider 

Format 

%  of  flights 

Minutes 

1-5 

1-5 

~~~~~ _ 

used  on 

spent 

Commercial  vendor 

Internet 

Wx  via  internet,  paid  (Which  site?  — L__ 

_____ 

DUATS 

Internet 

FAA  Direct  User  Access  Terminal 

DUATS 

at  airport 

FSS 

telephone 

Flight  Service  Station,  automated  briefing  (TIBS) 

FSS 

telephone 

FSS  standard  briefing 

FSS 

telephone 

FSS,  abbreviated  briefing 

FSS 

telephone 

FSS,  outlook  briefing 

The  Weather  Channel 

Internet, TV 

Cable  TV  weather 

Other  sources 

List 
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19.  The  same  way  you  did  in  Q18,  evaluate  the  top  8  preflight  weather  products  you  use  most  in  planning  a 
cross-country,  bad-wx  flight.  “Text”  format  means  sources  you  read  yourself  or  that  are  read  to  you. 

a.  Rank  ONLY  your  8  most-used  products.  Write  “1”  next  to  the  source  you  use  most,  etc. 

b.  Rate  the  value  each  of  these  8  using  the  1-5  scale  of  Q18b,  for  its  information  value. 

c.  Estimate  the  percentage  of  cross-country,  bad-wx  flights  during  which  you  used  each  of  the  8. 

d.  Estimate  the  average  number  of  minutes  spent  on  each  of  the  8  during  bad-wx  preflight. 


Rank 

Value 

Product 

Format 

Details 

%  of  flights 

Minutes 

1-8 

1-5 

used  on 

spent 

AC 

text 

Severe  Wx  Outlook  Narrative  (2-day  convective  outlook) 

AIRMET/SIGMET 

text 

Icing,  turbulence,  IFR,  convective  advisories,  watches 

ASOS 

radio 

Automated  Surface  Observing  System 

ATIS 

radio 

Automated  Terminal  Information  Service 

AWOS 

radio 

Automated  Weather  Observing  System 

charts,  Air-  or  Surface-analysis 

graphic 

Constant-pressure  (isobar)  charts 

charts,  Convective  outlook 

graphic 

48-hr  forecast  charts  for  T-storm  activity 

charts,  Prog. 

graphic 

12,  24-hr  prognostication  charts  w.  isobars,  wx  symbols 

charts,  Radar  (NEXRAD) 

graphic 

Doppler  radar  maps 

charts,  Radar  summary 

graphic 

Maps  of  precipitation  regions 

charts,  Weather  depiction 

graphic 

Maps  with  isobars,  precip,  IFR  regions,  ceilings 

FA 

text 

Aviation  area  18-hr  forecast 

FD 

text 

Winds  and  temps,  aloft  12-hr  forecast  charts 

FD 

graphic 

Winds  and  temps,  aloft  12-hr  forecast  charts 

GPS 

TorG 

Global  positioning  satellite 

LLWAS 

radio 

Low-Level  Wind  Shear  Alert  System  (at  airports) 

METAR 

text 

Meteorological  Aviation  Routine 

PIREP 

text 

Pilot  reports 

Satellite 

graphic 

Satellite  photos  of  cloud  cover 

SD 

text 

Radar  weather  reports  (hourly) 

TAF 

text 

Terminal  Aerodrome  Forecast 

TWEB 

text 

Transcribed  Weather  Broadcast  (over  telephone) 

WW,  AWW 

text 

Weather  Watch  bulletins,  severe 

Other  sources 

List 

20.  The  same  way  you  did  in  Q18,  evaluate  the  top  3  en  route  weather  sources  you  use  most  during  a  cross¬ 
country,  bad-weather  flight  (here,  a  “source”  can  either  be  a  product  or  a  provider). 

a.  Rank:  Rank  ONLY  your  3  most-used  sources.  Write  “1”  next  to  the  source  you  use  most ,  etc. 

b.  Value:  Using  the  1-5  scale  of  Q18b,  how  do  you  rate  each  of  these  3  source’s  information  value? 

c.  %:  Estimate  the  percentage  of  cross-country,  bad-weather  flights  you  use  these  3  sources  on. 

d.  Minutes:  Estimate  the  average  number  of  minutes  you  spend  on  each  during  bad-wx  flight. 


Rank 

Value 

Source 

Details 

%  of  flights 

Minutes 

1-3 

1-5 

used  on 

spent 

avionics 

(e.g.  on-board  radar,  Stormscope,  etc)  List 

ASOS 

Automated  Surface  Observing  System 

ATIS 

Automated  Terminal  Information  Service 

AWOS 

Automated  Weather  Observing  System 

EFAS 

Enroute  Flight  Advisory  System  (Flight  Watch  through  FSS) 

HI  WAS 

Hazardous  Inflight  Weather  Advisory  System  (selected  VORs) 

TWEB 

Transcribed  Weather  Broadcast  (over  VOR,  NDB) 

Other  sources 

List 

21.  Are  there  reasons  why  the  preflight  and  enroute  sources  you  USE  most  aren’t  the  ones  you  VALUE  most? 
If  so,  why?  (For  example,  some  of  the  graphic  Internet  products  download  slowly  on  a  modem.  Or  some 
products  may  be  unavailable.  Or  you  might  consider  some  too  incomplete  or  unreliable). 


22.  What  percentage  of  FSS  briefers  do  you  think  are  National  Weather  Service-certified?  (best  guess) 

23.  What  percentage  do  you  think  are  pilots? _ 


24.  Would  it  matter  to  you  if  your  briefer  were  not  a  pilot,  as  long  as  he/she  were  NWS-certified?  (circle 
answer) 

1  2  3  4  5 

not  at  all  a  little  bit  somewhat  quite  a  bit  an  extreme  amount 

25.  If  you  use  FSS  weather  briefings,  how  satisfied  are  you  with  them?  (leave  blank  if  you  don’t  use  FSS) 

1  2  3  4  5 

not  at  all  a  little  bit  somewhat  quite  satisfied  extremely 

satisfied 

26.  What  is  the  typical  number  of  weather  reporting  stations  (e.g.  KOKC,  KDWF)  you  check  before  an  average 

4-hour,  bad-weather  GA  flight? _  The  smallest  number? _  The  largest  number? _ 

Regarding  VFR  LOCAL  FLIGFIT,  what  percentage  of  the  time  do  you  do  the  following  (0-100%)? 

27.  I  get  a  briefing  on  the  weather  before  I  take  off .  . 

28.  1  request  weather  updates  during  flight .  . 

Regarding  VFR  CROSS-COUNTRY  FLIGFIT,  what  percentage  of  the  time  do  you  do  the  following? 

29.  I  get  a  briefing  on  the  weather  before  I  take  off .  . 

30.  I  request  weather  updates  for  route  &  destination  during  flight  _ 

Answer  questions  31  through  34  using  a  scale  of  “0”  through  “6  or  more”:  How  many  times  have  you  . . . 

31.  become  so  disoriented  that  you  had  to  land  or  call  ATC  for  assistance  in  determining  your  location? 

32.  flown  into  areas  of  IMC  without  an  instrument  rating  or  an  instrument-qualified  aircraft? . 

33.  become  so  disoriented  after  entering  IMC  that  you  had  difficulty  in  maintaining  aircraft  control? 

34.  turned  back  or  diverted  to  another  airport  because  of  bad  weather  while  on  a  VFR  flight? . 


Use  the  scale  below  to  answer  Qs  35-38 


1 

2 

3 

4 

5 

not  at  all 

a  little  bit 

somewhat 

quite  a  bit 

an  extreme  amount 

35.  How  much  does  the  distance  you  have  to  fly  through  bad  weather  affect  your  willingness  to  fly? 

36.  Does  having  non-family  passengers  affect  your  willingness  to  fly  in  bad  weather? . 

37.  Does  having  family  passengers  affect  your  willingness  to  fly  in  bad  weather? . 

38.  Has  social  or  corporate  pressure  ever  affected  your  willingness  to  fly  in  bad  weather? . 

39.  Have  you  ever  had  a  life-threatening  flight  experience  related  to  weather?  (Y  /  N) . 

(On  Q  39,  if  answer  is  3,  4,  or  5,  please  briefly  describe  your  experiences). 

THIS  CONCLUDES  THE  SUBJECT  MATTER  EXPERT  INTERVIEW.  THANKS  AGAIN. 
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APPENDIX  B 


(Top)  Coding  rubric  for  qualitative  analysis,  with  frequencies  of  incidence. 
(Bottom)  Deductive  versus  inductive  elements  of  the  rubric. 


I 


][ 


Theory-driven  elements 

Data-driven  elements 

(Deductive) 

(Inductive) 
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